Papers with model size
Structured Pruning for Large Language Models Using Coupled Components Elimination and Minor Fine-tuning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated powerful capabilities in natural language processing, yet their vast number of parameters poses challenges for deployment and inference efficiency. |
| Approach: | They propose a structured pruning algorithm that derives the importance of different components based on intermediate data dependencies and removes coupled components across different layers simultaneously. |
| Outcome: | The proposed algorithm reduces model size and accelerates inference without specialized operators and libraries, while maintaining its utility as versatile problem solvers. |
STAND-Guard: A Small Task-Adaptive Content Moderation Model (2025.coling-industry)
Copied to clipboard
| Challenge: | Content moderation is important for developing welcoming online platforms and responsible large language models. |
| Approach: | They propose a small task-adaptive coNtent moDeration model that can be easily adapted to new or customized content moderation tasks without extensive model tuning. |
| Outcome: | The proposed model is comparable to GPT-3.5-Turbo on unseen English binary classification tasks. |
Bag-of-Words Transfer: Non-Contextual Techniques for Multi-Task Learning (D19-61)
Copied to clipboard
| Challenge: | Existing approaches to multi-task learning take advantage of transfer among tasks . generative reconstruction of the observations is not included in the standard framework . |
| Approach: | They propose to use a syntactically-oblivious pooling encoder and pre-trained word embeddings to improve sentence-level representations. |
| Outcome: | The proposed techniques yield similar performance on a universe of task combinations while reducing training time and model size. |
Scaling Parameter-Constrained Language Models with Quality Data (2024.emnlp-industry)
Copied to clipboard
Ernie Chang, Matteo Paltenghi, Yang Li, Pin-Jie Lin, Changsheng Zhao, Patrick Huber, Zechun Liu, Rastislav Rabatin, Yangyang Shi, Vikas Chandra
| Challenge: | Scaling laws in language modeling quantify training loss as a function of dataset size and model parameters, but neglect the critical role of data quality in model generalization. |
| Approach: | They propose to use effective training tokens as a combination of text diversity and syntheticity as measured by a teacher model to calculate scaling laws. |
| Outcome: | The proposed term effective training tokens is a combination of two readily-computed indicators of text diversity and syntheticity as measured by a teacher model. |
Natural Language Generation by Hierarchical Decoding with Linguistic Patterns (N18-2)
Copied to clipboard
| Challenge: | a common and mostly adopted method is the rule-based (or template-based) method for natural language generation. |
| Approach: | They propose a hierarchical decoding NLG model based on linguistic patterns in different levels. |
| Outcome: | The proposed method outperforms the traditional one with a smaller model size. |
How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2024.acl-long)
Copied to clipboard
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, Jingren Zhou
| Challenge: | supervised fine-tuning (SFT) is a technique used to enhance multiple abilities in large language models. |
| Approach: | They propose to study the interplay of data composition between mathematical reasoning, code generation, and general human-aligning abilities during supervised fine-tuning. |
| Outcome: | The proposed model improves math reasoning and code generation with increasing data amount . the proposed model size and SFT strategies can be used to learn multiple skills with different scaling patterns. |
Cooperative Self-training of Machine Reading Comprehension (2022.naacl-main)
Copied to clipboard
| Challenge: | Pretrained language models provide high-quality contextualized word embeddings, but training question answering models requires large amounts of annotated data for specific domains. |
| Approach: | They propose a framework for automatically generating more non-trivial question-answer pairs to improve model performance. |
| Outcome: | The proposed framework outperforms state-of-the-art (SOTA) pretrained language models and transfer learning approaches on standard question-answering benchmarks. |
Retrieve-and-Fill for Scenario-based Task-Oriented Semantic Parsing (2023.eacl-main)
Copied to clipboard
Akshat Shrivastava, Shrey Desai, Anchit Gupta, Ali Elkahky, Aleksandr Livshits, Alexander Zotov, Ahmed Aly
| Challenge: | Task-oriented semantic parsing models have achieved strong results in recent years, but they often face obstacles adapting to novel settings with distinct semantics and scarce data. |
| Approach: | They propose a scenario-based semantic parsing model which isolates coarse-grained and fine-grounded aspects of the task and solves them with off-the-shelf neural modules. |
| Outcome: | The proposed model outperforms previous approaches in high-resource, low-resourced, and multilingual settings, and is modular, differentiable, interpretable, and allows extra supervision from scenarios. |
Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement (2026.eacl-industry)
Copied to clipboard
Aaditya Shukla, Sidney Knowles, Meenakshi Madugula, David Farris, Ryan Angilly, Santiago Pombo, Lu An, Anbang Xu, Abhinav Balasubramanian, Tan Yu, Jiaxiang Ren, Rama Akkiraju
| Challenge: | NVInfo AI is a generative AI agent that can be deployed in production without full-scale retraining or infrastructure overhauls. |
| Approach: | They propose to implement a retrieval-augmented generation (RAG)-driven data flywheel in NVInfo AI, a mixture-of-experts knowledge assistant, for 30,000 employees. |
| Outcome: | The proposed system addresses failures in retrieval-augmented generation pipelines and enables continuous learning. |
MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to embed multimodal models face limitations such as suboptimal causal attention in VLMs and limited diversity in training objectives and data. |
| Approach: | They propose a framework for transforming pre-trained VLMs into bidirectional multimodal embedding models. |
| Outcome: | The proposed model improves performance across MMEB and ViDoRe-v2 benchmarks and exhibits strong scalability with both model size and training data on MMEF. |
BMCook: A Task-agnostic Compression Toolkit for Big Models (2022.emnlp-demos)
Copied to clipboard
Zhengyan Zhang, Baitao Gong, Yingfa Chen, Xu Han, Guoyang Zeng, Weilin Zhao, Yanxu Chen, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing efforts to compress medium-sized models for specific tasks have limited results. |
| Approach: | They propose a task-agnostic compression toolkit for big models that implements quantization, pruning, distillation and MoEfication methods. |
| Outcome: | The proposed tool improves performance on a model with 3 billion parameters by 12x . it also outperforms the original model on three typical NLP benchmarks. |
Fast Vocabulary Transfer for Language Model Compression (2022.emnlp-industry)
Copied to clipboard
| Challenge: | Existing methods to reduce model size and size are expensive and inefficient for some applications. |
| Approach: | They propose a method that relies on vocabulary transfer to reduce model size and inference time while compromising on performance. |
| Outcome: | The proposed method reduces model size and inference time while compromising on performance. |
EELBERT: Tiny Models through Dynamic Embeddings (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Empirical evaluation shows that the input embedding layer occupies a large portion of the model size. |
| Approach: | They propose an approach for compression of transformer-based models with minimal impact on downstream tasks by replacing the input embedding layer with dynamic embeddable computations. |
| Outcome: | Empirical evaluation shows that the proposed model is 15x smaller (1.2 MB) compared to the traditional model. |
Personal Information Parroting in Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Modern language models memorize millions of PI instances, increasing privacy risks. |
| Approach: | They develop a model that parrots 13.6% of PI verbatim on a manually curated set of 483 instances . they recommend that pretraining datasets be aggressively filtered and anonymized to minimize PI parroting. |
| Outcome: | The proposed model outperforms the best regex-based PI detectors on a manually curated set of 483 instances of PI. |
CoLLiE: Collaborative Training of Large Language Models in an Efficient Way (2023.emnlp-demo)
Copied to clipboard
Kai Lv, Shuo Zhang, Tianle Gu, Shuhao Xing, Jiawei Hong, Keyu Chen, Xiaoran Liu, Yuqing Yang, Honglin Guo, Tengxiao Liu, Yu Sun, Qipeng Guo, Hang Yan, Xipeng Qiu
| Challenge: | Large language models (LLMs) are increasingly pivotal in a wide range of tasks . however, the resources required for training these models necessitate efficient solutions . |
| Approach: | They propose a library that facilitates collaborative training of large language models . they use 3D parallelism, parameter-efficient fine-tuning methods and optimizers . |
| Outcome: | The proposed library has proven superior training efficiency in comparison with prevalent solutions in pre-training and fine-tuning scenarios. |
Speed Without Sacrifice: Fine-Tuning Language Models with Medusa and Knowledge Distillation in Travel Applications (2025.acl-industry)
Copied to clipboard
Daniel Zagyva, Emmanouil Stergiadis, Laurens Van Der Maas, Aleksandra Dokic, Eran Fainman, Ilya Gusev, Moran Beladev
| Challenge: | Rapid growth of digital applications has intensified the demand for real-time natural language processing (NLP) capabilities. |
| Approach: | They propose a framework that combines Medusa and knowledge distillation to achieve compounded benefits in both model size and inference speed. |
| Outcome: | The proposed framework reduces inference latency by 10-20x while maintaining the student model’s performance quality. |
Being Right for Whose Right Reasons? (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work has failed to acknowledge that what counts as a rationale is subjective. |
| Approach: | They propose to use demographic annotations to augment existing datasets to ask what demographics our models align with and whose reasoning patterns they align with. |
| Outcome: | The proposed model rationales align better with older and/or white annotators, and are biased towards older and white anorators. |
Learning High-Quality and General-Purpose Phrase Representations (2024.findings-eacl)
Copied to clipboard
| Challenge: | Pre-trained language models for phrasal embeddings are unnecessarily complex and require to be pre-tuned on a corpus with context sentences. |
| Approach: | They propose a framework to learn phrase representations in a context-free fashion. |
| Outcome: | The proposed framework generates superior phrase embeddings while requiring a smaller model size. |
Building an Efficient Multilingual Non-Profit IR System for the Islamic Domain Leveraging Multiprocessing Design in Rust (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Existing models that are pre-trained on a general domain can deteriorate performance due to domain shift when applied to new domains. |
| Approach: | They propose to train a multilingual non-profit IR system for the Islamic domain using Rust Language capabilities. |
| Outcome: | The proposed model outperforms models pre-trained on general domains and on resource-constrained devices. |
Wait, that’s not an option: LLMs Robustness with Incorrect Multiple-Choice Options (2025.acl-long)
Copied to clipboard
| Challenge: | Using a framework that combines instruction-following with critical reasoning, we show that the ability of LLMs to override defaults when faced with invalid options is impaired by alignment techniques. |
| Approach: | They propose a framework for evaluating LLMs’ capacity to balance instruction-following with critical reasoning when presented with multiple-choice questions containing no valid answers. |
| Outcome: | The proposed framework improves models' ability to override defaults when faced with invalid options while minimizing the impact of model size and training techniques on the model. |
Using Commonsense Knowledge to Answer Why-Questions (2022.emnlp-main)
Copied to clipboard
Yash Kumar Lal, Niket Tandon, Tanvi Aggarwal, Horace Liu, Nathanael Chambers, Raymond Mooney, Niranjan Balasubramanian
| Challenge: | Existing approaches to integrating commonsense knowledge into large language models are implicit and explicit. |
| Approach: | They analyze the effects of model size and methods of injecting knowledge into TellMeWhy datasets to determine what aspects of commonsense knowledge are available in large language models. |
| Outcome: | The largest models yield substantial improvements over base models, but the amount of improvement decreases with larger model size. |
How much do contextualized representations encode long-range context? (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies of contextualized representations focus on short sequences of tens to hundreds of tokens, whereas modern language models handle hundreds of thousands of token in a single context window. |
| Approach: | They use a perturbation setup and a metric to capture contextualization of long-range patterns from the perspective of representation geometry. |
| Outcome: | The proposed model can encode long-range contexts, but it's not fully recurrent, the authors say . their results suggest improvements in existing language models . |
Pruning as a Domain-specific LLM Extractor (2024.findings-naacl)
Copied to clipboard
Nan Zhang, Yanchi Liu, Xujiang Zhao, Wei Cheng, Runxue Bao, Rui Zhang, Prasenjit Mitra, Haifeng Chen
| Challenge: | Large Language Models (LLMs) have exhibited remarkable proficiency across a wide array of NLP tasks. |
| Approach: | They propose a method for pruning large language models using general or task-specific weights to extract a compressed, task-agnostic LLM. |
| Outcome: | The proposed method extracts a compressed, domain-specific, and task- agnostic LLM by identifying LLM weights that are pivotal for general capabilities, like linguistic capability and multi-task solving, and domain- specific knowledge. |
Harnessing PDF Data for Improving Japanese Large Multimodal Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Multimodal Models (LMMs) have demonstrated strong performance in English, but their effectiveness in Japanese remains limited due to the lack of high-quality training data. |
| Approach: | They propose a pipeline that leverages pretrained models to extract image-text pairs from PDFs . they use layout analysis, OCR, and vision-language pairing to enrich the training data . |
| Outcome: | The proposed pipeline extracts image-text pairs from Japanese PDFs, eliminating manual annotations. |
Cognitive Biases, Task Complexity, and Result Intepretability in Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Recent work shows that cognitive biases occur frequently in language models . a cognitive bias is a systematic deviation in judgment that simplifies complex decisions . |
| Approach: | They evaluate the performance of different groups of models for each type of cognitive bias . they find that task complexity plays a part in eliciting stronger effects for some biases . |
| Outcome: | The proposed models perform better for each type of bias in different settings . the results show that task complexity plays a part in eliciting stronger effects . |
A Progressive Model to Enable Continual Learning for Semantic Slot Filling (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to slot filling training on large scale data are inefficient and require multiple trainings. |
| Approach: | They propose a slot filling model that transfers previously learned knowledge to a small size expanded component and enables it to be fast trained to learn from new data. |
| Outcome: | The proposed model outperforms existing models on two benchmark datasets by 4.24% and 3.03% on the same dataset. |
Deploying Tiny LVLM Judges for Real-World Evaluation of Chart Models: Lessons Learned and Best Practices (2025.emnlp-industry)
Copied to clipboard
Md Tahmid Rahman Laskar, Mohammed Saidul Islam, Ridwan Mahbub, Mizanur Rahman, Amran Bhuiyan, Israt Jahan, Mir Tafseer Nayeem, Shafiq Joty, Enamul Hoque, Jimmy Huang
| Challenge: | Large Vision-Language Models (LVLMs) with only 7B parameters perform poorly as judges in resource-constrained settings. |
| Approach: | They propose two approaches to ensure costefficient evaluation by combining multiple criteria into a single query and domainadaptive transfer learning to create a 2Bparameter VLM on a chart dataset. |
| Outcome: | The proposed model can effectively transfer knowledge from one dataset to another to make it a more specialized model. |
Adversarial Robustness of Prompt-based Few-Shot Learning for Natural Language Understanding (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent few-shot learning methods focus on improving downstream task performance, but there is limited understanding of the adversarial robustness of such methods. |
| Approach: | They evaluate prompt-based FSL methods against fully fine-tuned models to better understand the impact of various factors towards robustness. |
| Outcome: | The proposed methods show that they are less robust in the face of adversarial perturbations than fully fine-tuned models. |
Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models (2022.emnlp-main)
Copied to clipboard
| Challenge: | a new method for textual style transfer is proposed for text with a limited set of style choices . textual styles are a complex task that requires specialized models to perform . |
| Approach: | They propose a method for arbitrary textual style transfer using pre-trained language models . they use a mathematical formulation of the TST task, decomposing it into three components . |
| Outcome: | The proposed method performs on par with state-of-the-art large-scale models while using less compute and memory. |
E-Bench: Towards Evaluating the Ease-of-Use of Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | E-Bench is a framework for easy-to-use research on large language models. |
| Approach: | They propose to evaluate the ease-of-use of large language models and construct an E-Bench . they simulate human use from synonymous and typographical perturbations . |
| Outcome: | The proposed model is able to resist synonymous expressions and typos and improves performance. |
Frequency Explains the Inverse Correlation of Large Language Models’ Size, Training Data Amount, and Surprisal’s Fit to Reading Times (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that as Transformer-based language models become larger and are trained on very large amounts of data, the fit of their surprisal estimates to naturalistic human reading times degrades. |
| Approach: | They present a series of analyses showing that word frequency is a key explanatory factor underlying these two trends. |
| Outcome: | The results show that word frequency is a key explanatory factor underlying these two trends. |
Towards an On-device Agent for Text Rewriting (2024.findings-naacl)
Copied to clipboard
Yun Zhu, Yinxiao Liu, Felix Stahlberg, Shankar Kumar, Yu-Hui Chen, Liangchen Luo, Lei Shu, Renjie Liu, Jindong Chen, Lei Meng
| Challenge: | Large Language Models (LLMs) have demonstrated impressive capabilities for text rewriting, however creating a smaller yet potent language model presents two formidable challenges: costly data collection and absence of emergent capabilities. |
| Approach: | They propose a new instruction tuning method to develop a mo-bile text rewriting model that leverages LLM-generated data and heuristic reinforcement learning, eliminating the need for human data collection. |
| Outcome: | The proposed model surpasses the current state-of-the-art LLMs in text rewriting while maintaining a significantly reduced model size using public benchmark EditEval and our new benchmark. |
With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models and Vision Language Model (VLMs) have demonstrated aptitude as potential substitutes for human participants in psycholinguistic experiments. |
| Approach: | They examine whether large language models and vision language models implicitly understand sound-based phenomena via orthography and imagery alone. |
| Outcome: | The proposed models demonstrate sound symbolism and ability to "hear" using language and vision modules. |
DeSIQ: Towards an Unbiased, Challenging Benchmark for Social Intelligence Understanding (2023.emnlp-main)
Copied to clipboard
| Challenge: | Social intelligence is essential for understanding and reasoning about human expressions, intents and interactions. |
| Approach: | They propose a methodology to study the soundness of Social-IQ by applying simple perturbations to a dataset of multiple choice questions on videos of complex social interactions. |
| Outcome: | The proposed method reduces biases in the original dataset and improves performance. |
Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large pre-trained models have achieved outstanding results in sequence modeling . alternative architectures, such as Selective Structured State Space Models (SSMs), have been proposed to address these inefficiencies. |
| Approach: | They propose to reduce the size and computational overhead of large pre-trained models by removing selected components at different granularities. |
| Outcome: | The proposed models achieve a speedup of up to 1.4x during inference while maintaining accuracy. |
Multi-stage Pre-training over Simplified Multimodal Pre-training Models (2021.acl-long)
Copied to clipboard
| Challenge: | Existing multimodal pre-training models require large amounts of training data and have huge model sizes, making them impossible to apply in low-resource situations. |
| Approach: | They propose a multi-stage pre-training method which uses information at different granularities from word, phrase to sentence in both texts and images to pre-train a model in stages. |
| Outcome: | The proposed method outperforms the original model in Image-Text Retrieval task and outperformed the original LXMERT model in downstream tasks. |
Demystifying Mixed Outcomes of Self-Training: Pre-training Analyses on Non-Toy LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Recent studies on self-training report seemingly contradictory outcomes. |
| Approach: | They use OLMo-2 models as non-toy LLMs and perform multiple rounds of continual pre-training using self-generated text with different prompting strategies and data filtering. |
| Outcome: | The proposed model collapse is inherent to the training procedure itself, while self-improvement is likely owes its success to human-designed, strategic synthetic pipelines that inject external intelligence. |
Faithful Question Answering with Monte-Carlo Planning (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to answer questions using large language models lack the ability to faithfully follow the intermediate reasoning steps from the known premises to the answer. |
| Approach: | They propose a faithful question-answering task that uses a Monte-Carlo planning algorithm to produce faithful reasoning steps from the known premises to the answer. |
| Outcome: | The proposed task can produce valid and faithful reasoning steps compared with large language models with a much smaller model size. |
Scaling Laws for Multilingual Language Models (2025.findings-acl)
Copied to clipboard
Yifei He, Alon Benhaim, Barun Patra, Praneetha Vaddamanu, Sanchit Ahuja, Parul Chopra, Vishrav Chaudhary, Han Zhao, Xia Song
| Challenge: | Existing scaling laws for language models are limited to a limited number of languages, but they can be applied to arbitrary number of different languages. |
| Approach: | They propose a scaling law for general-purpose decoder-only language models trained on multilingual data that shifts focus from individual languages to language families. |
| Outcome: | The proposed scaling law can be applied to models trained on multilingual data . it can be used to predict performance across multiple languages and models . |
SMART: Self-Aware Agent for Tool Overuse Mitigation (2025.findings-acl)
Copied to clipboard
Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Tür, Gokhan Tur, Heng Ji
| Challenge: | Current Large Language Models (LLMs) lack self-awareness to balance reasoning and tool use, increasing computational overhead. |
| Approach: | They propose a paradigm that enhances an agent’s self-awareness to optimize task handling and reduce tool overuse. |
| Outcome: | The proposed model reduces tool use by 24% while improving performance by over 37%. |
QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models (2025.findings-naacl)
Copied to clipboard
| Challenge: | Structured pruning can reduce model size but results in significant accuracy degradation . quantization and pruning increase the difficulty of fine-tuning, requiring a more refined quantization scheme. |
| Approach: | They propose a structured pruning framework followed by a layer-wise mixed-precision quantization scheme to reduce model memory consumption during fine-tuning and inference. |
| Outcome: | Experiments on benchmark datasets show that QPruner outperforms existing methods in memory savings while maintaining or improving model performance. |
Empirical Study of Zero-shot Keyphrase Extraction with Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | a prompting-based approach can effectively supersede traditional KE methods, a study shows . our code is available at https://github.com/kangnlp/zero-shot-keyphrase-extraction-with-LLMs. |
| Approach: | They propose four prompting strategies for zero-shot keyphrase extraction using Large Language Models. |
| Outcome: | The proposed prompting strategies outperform state-of-the-art prompting methods on KE benchmark datasets. |
HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large pre-trained language models (PLMs) have shown overwhelming performances on many tasks, but their large size and slow inference speed have hindered practical deployments. |
| Approach: | They propose a hierarchical relational knowledge distillation method to capture hierarchic and domain relational information. |
| Outcome: | The proposed method outperforms existing methods on multi-domain datasets and is highly reproducible. |
Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks (2022.coling-1)
Copied to clipboard
| Challenge: | Quantization, knowledge distillation, and magnitude pruning are among the most popular methods for neural network compression in NLP. |
| Approach: | They compare accuracy vs. model size tradeoffs using quantization and distillation methods . they find that pruning provides greater benefit than quantization . |
| Outcome: | The proposed methods reduce model size and can accelerate inference, but their relative benefit and combinatorial interactions have not been rigorously studied. |
QueryForm: A Simple Zero-shot Form Entity Query Framework (2023.findings-acl)
Copied to clipboard
Zifeng Wang, Zizhao Zhang, Jacob Devlin, Chen-Yu Lee, Guolong Su, Hao Zhang, Jennifer Dy, Vincent Perot, Tomas Pfister
| Challenge: | Form-like document understanding is a key yet under-investigated problem . endlessly training specialized models on new document types is not scalable in many practical scenarios. |
| Approach: | They propose to use large-scale query-entity pairs generated from form-like webpages to pre-train QueryForm. |
| Outcome: | The proposed framework sets state-of-the-art average F1 score on XFUND and Payment benchmarks. |
FactCG: Enhancing Fact Checkers with Graph-Based Multi-Hop Data (2025.naacl-long)
Copied to clipboard
| Challenge: | Prior research on training grounded factuality classification models to detect hallucinations in large language models (LLMs) has relied on public natural language inference (NLI) data and synthetic data. |
| Approach: | They propose a method that leverages multi-hop reasoning on context graphs extracted from documents to generate complex multi-level claims without relying on LLMs to decide data labels. |
| Outcome: | The proposed model outperforms GPT-4-o on the LLM-Aggrefact benchmark with much smaller model size. |
The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models (2022.emnlp-main)
Copied to clipboard
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, Dan Alistarh
| Challenge: | Pre-trained Transformer models provide robust language representations which can be specialized on various tasks. |
| Approach: | They propose an efficient pruning method based on approximate second-order information that allows pruning weight blocks to be used for pruning. |
| Outcome: | The proposed method is the first to be applied at the BERT scale and significantly pushes the boundaries of the current sparse models with respect to all metrics: model size, inference speed and task accuracy. |
QUENCH: Measuring the gap between Indic and Non-Indic Contextual General Reasoning in LLMs (2025.coling-main)
Copied to clipboard
| Challenge: | QUENCH is a text-based English quizzing benchmarking system for large language models (LLMs). |
| Approach: | They propose a text-based English Quizzing Benchmark manually curated from YouTube quiz videos. |
| Outcome: | The proposed system assesses the world knowledge and deduction capabilities of large language models via a zero-shot, open-domain quizzing setup. |
Structure-Level Knowledge Distillation For Multilingual Sequence Labeling (2020.acl-main)
Copied to clipboard
| Challenge: | Existing multilingual models still underperform individual monolingual models due to model capacity limitations. |
| Approach: | They propose to distill the structural knowledge of several monolingual models (teachers) to the unified multilingual model (student). |
| Outcome: | The proposed model outperforms strong baseline models and teacher models on 4 multilingual tasks with 25 datasets and has stronger zero-shot generalizability. |
Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | Negation is a fundamental operation in natural language that reverses the meaning of an expression into its opposite. |
| Approach: | They propose a sentence-level negation understanding benchmark that measures negation performance in Korean. |
| Outcome: | The proposed benchmark improves negation understanding and broader comprehension in Korean. |
On the Multilingual Capabilities of Very Large-Scale English Language Models (2022.lrec-1)
Copied to clipboard
| Challenge: | Generative Pre-trained Transformers (GPTs) have been scaled to unprecedented sizes in the history of machine learning. |
| Approach: | They investigate the potential and limits of Generative Pre-trained Transformers in three tasks . they find it can be almost as useful for many languages as it is for English . |
| Outcome: | The proposed model can perform tasks in five different languages, and its potential is explored . it can learn from a few examples "via text interaction" and is scalable to many languages . |
CCIM: Cross-modal Cross-lingual Interactive Image Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on text image machine translation (TIMT) lacks recognized source language information resulting in a decrease in translation performance. |
| Approach: | They propose a cross-modal cross-lingual interactive model which incorporates source language information by synchronizing source and target language results. |
| Outcome: | The proposed model outperforms end-to-end models and has faster decoding speed with smaller model size than cascade models. |
Multilingual Constituency Parsing with Self-Attention and Pre-Training (P19-1)
Copied to clipboard
| Challenge: | a range of pre-training conditions can be used for constituency parsing, but large model sizes make it expensive to train separate models for each language. |
| Approach: | They compare the benefits of no pre-training, fastText, ELMo, and BERT for English . they also find that pre- training is beneficial across all 11 languages tested . |
| Outcome: | The proposed model outperforms fastText, ELMo, and BERT for English . but large model sizes make it expensive to train separate models for each language . |
DRAG: Distilling RAG for SLMs from LLMs to Transfer Knowledge and Mitigate Hallucination via Evidence and Graph-based Distillation (2025.acl-long)
Copied to clipboard
Jennifer Chen, Aidar Myrzakhan, Yaxin Luo, Hassaan Muhammad Khan, Sondos Mahmoud Bsharat, Zhiqiang Shen
| Challenge: | Large-scale RAG systems consume significant computational resources and are prone to generating “hallucinated” content from Humans. |
| Approach: | They propose a framework for distilling RAG knowledge from large-scale language models into small LMs. |
| Outcome: | The proposed method outperforms the prior competitive RAG methods like MiniRAG for SLMs by up to 27.7% using the same models, preserving high-level efficiency and reliability. |
On-device Structured and Context Partitioned Projection Networks (P19-1)
Copied to clipboard
| Challenge: | A challenge in on-device text classification is to build highly accurate models that fit in small memory footprint and have low latency. |
| Approach: | They propose an on-device neural network which learns compact projection vectors from raw text using structured and context-dependent partition projections. |
| Outcome: | The proposed model outperforms baseline models and surpasses RNN, CNN and BiLSTM models on dialog act and intent prediction. |
PersonaGym: Evaluating Persona Agents and LLMs (2025.findings-emnlp)
Copied to clipboard
Vinay Samuel, Henry Peng Zou, Yue Zhou, Shreyas Chaudhari, Ashwin Kalyan, Tanmay Rajpurohit, Ameet Deshpande, Karthik R Narasimhan, Vishvak Murahari
| Challenge: | Persona agents are LLM agents conditioned to act according to an assigned persona . evaluating how faithfully these agents adhere to their personas remains a challenge . |
| Approach: | a new study evaluates persona agents' ability to act according to an assigned persona . a persona agent's person score is a human-aligned automatic metric that can be used to evaluate a model . |
| Outcome: | a new evaluation framework and a human-aligned automatic metric show that persona agents can perform better. |
BioMegatron: Larger Biomedical Domain Language Model (2020.emnlp-main)
Copied to clipboard
Hoo-Chang Shin, Yang Zhang, Evelina Bakhturina, Raul Puri, Mostofa Patwary, Mohammad Shoeybi, Raghav Mani
| Challenge: | Existing studies on domain language models do not study the factors affecting performance on domain languages. |
| Approach: | They empirically evaluate factors that can affect performance on domain language applications . sub-word vocabulary set, model size, pre-training corpus, and domain transfer are important . |
| Outcome: | The results show language models trained on biomedical text perform better on biomedicine benchmarks than those trained on general domain text corpora. |
Deep Model Compression Also Helps Models Capture Ambiguity (2023.acl-long)
Copied to clipboard
| Challenge: | Experimental results show that our method reduces the model size significantly and improves latency. |
| Approach: | They propose a method to capture the degree of relationship between a sample and its candidate classes by deep model compression. |
| Outcome: | The proposed method reduces model size significantly and improves latency. |
NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Structured pruning methods have proven effective in reducing the model size and accelerating inference speed in various network architectures. |
| Approach: | They propose a framework that narrows the encoder and shortens the decoder networks of encoder-decoder models. |
| Outcome: | The proposed framework reduces the number of decoder layers and improves generation quality. |
EIT: Enhanced Interactive Transformer (2024.acl-long)
Copied to clipboard
| Challenge: | Existing multi-view learning models prioritize complementarity while ignoring consensus . EMHA allows for efficient modeling of global dependencies among tokens in parallel . |
| Approach: | They propose an enhanced multi-head self-attention (EMHA) that prioritizes complementarity while ignoring consensus. |
| Outcome: | The proposed method favors consensus among heads by introducing two models . it is superior on a wide range of language tasks with a modest increase in model size . |
FedSpaLLM: Federated Pruning of Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing pruning methods assume public access to calibration data, which is impractical for privacy-sensitive applications. |
| Approach: | They propose a federated learning framework for pruning LLMs that prunes models locally based on private data while accounting for system heterogeneity and communication efficiency. |
| Outcome: | The proposed framework reduces communication overhead and personalizes pruning process based on client resources in federated settings. |
On the Analysis and Distillation of Emergent Outlier Properties in Pre-trained Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies show that a small subset of dimensions within language Transformers’ representation spaces emerge as "outliers" during pretraining. |
| Approach: | They propose a method that prioritizes critical outlier dimensions in distillation using a weighted MSE loss. |
| Outcome: | The proposed method outperforms state-of-the-art distillation methods and generalizes well across Encoder-only BERT, Decoder-only GPT-2, and Encodeer-Decoder T5 architectures. |
Extremely Low Bit Transformer Quantization for On-Device Neural Machine Translation (2020.findings-emnlp)
Copied to clipboard
Insoo Chung, Byeongwook Kim, Yoonjung Choi, Se Jung Kwon, Yongkweon Jeon, Baeseong Park, Sangha Kim, Dongsoo Lee
| Challenge: | Quantization is an effective technique to address heavy computation load and memory overhead during inference. |
| Approach: | They propose a low-bit quantization strategy to represent Transformer weights by an extremely low number of bits. |
| Outcome: | The proposed model achieves 11.8 smaller model size than baseline model, with less than -0.5 BLEU. |
Multilingual Sequence-to-Sequence Models for Hebrew NLP (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent work on pretrained language models for Hebrew is under-parameterized and under-trained . previous work on pretraining Hebrew LMs focused on encoder-only architectures . |
| Approach: | They propose to use sequence-to-sequence generative architectures to train large LMs in morphologically rich languages such as Hebrew. |
| Outcome: | The proposed model improves on all existing Hebrew NLP benchmarks. |
Towards Pareto-Efficient RLHF: Paying Attention to a Few High-Reward Samples with Reward Dropout (2024.findings-emnlp)
Copied to clipboard
| Challenge: | RLHF is a bi-objective problem that has the nature of a Pareto optimization . reward dropout is generalizable and most effective with non-pretrained target models . |
| Approach: | They propose a method that guarantees a Pareto improvement by leveraging reinforcement learning to fine-tune language models. |
| Outcome: | The proposed method guarantees a Pareto improvement on two benchmark datasets . it is generalizable and most effective with non-pretrained target models, saving the effort of pretraining. |
A Thorough Examination of Decoding Methods in the Era of LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Decoding methods are essential for converting language models from next-token predictors into practical task solvers. |
| Approach: | They propose to evaluate decoding methods in general-purpose large language models . they find that decoding method performance is notably task-dependent . |
| Outcome: | The proposed methods perform task-dependently and are influenced by alignment, model size, and quantization. |
CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing (2022.acl-long)
Copied to clipboard
| Challenge: | Existing work has resorted to sharing weights among models, but results are not affordable for real-world deployment. |
| Approach: | They propose a consistency-regularized ensemble learning approach based on perturbed models to retain ensemble benefits while maintaining a low memory cost. |
| Outcome: | The proposed approach outperforms the standard ensemble of 8 BERT-base models on the GLUE benchmark by 0.7 with a significantly smaller model size. |
GIMMICK: Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarking (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on Large Vision-Language Models (LVLMs) focus on a narrow range of cultures, focus on only a small number of cultural aspects or evaluate a limited selection of models on ONE task only. |
| Approach: | They propose a multimodal benchmark to assess a broad spectrum of cultural knowledge across 144 countries representing six global macro-regions. |
| Outcome: | The proposed benchmark examines cultural knowledge across 144 countries across six global macro-regions. |
The Cascade Transformer: an Application for Efficient Answer Sentence Selection (2020.acl-main)
Copied to clipboard
| Challenge: | Recent research shows that transformer-based neural networks can greatly advance the state of the art over many natural language processing tasks. |
| Approach: | They propose a technique to adapt transformer-based models into a cascade of rankers. |
| Outcome: | The proposed technique reduces computation by 37% with almost no impact on accuracy on two English question answering datasets. |
Towards Context-Robust LLMs: A Gated Representation Fine-tuning Approach (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) enhanced with external contexts face challenges in handling imperfect evidence. |
| Approach: | They propose a framework that can balance internal knowledge with external contexts . they propose gating mechanisms and low-rank representation adapters to adjust hidden representations based on a lightweight intervention function . |
| Outcome: | The proposed model can effectively balance internal knowledge with external context, similar to human cognitive processes. |
Sherry: Hardware-Efficient 1.25-Bit Ternary Quantization via Fine-grained Sparsification (2026.acl-long)
Copied to clipboard
| Challenge: | ternary quantization is a powerful solution for resource-constrained edge devices . current implementations suffer from a fundamental misalignment with commodity hardware . |
| Approach: | They propose a hardware-efficient ternary quantization framework that packs weights into five bits to restore power-of-two alignment. |
| Outcome: | The proposed framework reduces weights to -1, 0, +1 while preserving power-of-two alignment. |
Distilling Efficient Language-Specific Models for Cross-Lingual Transfer (2023.findings-acl)
Copied to clipboard
| Challenge: | Massively multilingual Transformers (MMTs) are widely used for cross-lingual transfer learning. |
| Approach: | They propose to extract compressed, language-specific models from MMTs which retain the capacity of the original MMT for cross-lingual transfer. |
| Outcome: | The proposed model outperforms models trained from scratch in zero-shot cross-lingual transfer across benchmarks. |
On Diversified Preferences of Large Language Model Alignment (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be fine tuned with human feedback, but human preferences can be diversified due to annotators’ different tastes, which hinders the effectiveness of LLM alignment methods. |
| Approach: | They propose a calibration error metric to evaluate large language models (LLMs) and a multi-objective reward learning method to enhance the calibration performance of RMs on shared preferences. |
| Outcome: | The proposed model can be adopted as a key calibration error and MORE can achieve superior alignment performance. |
Development of Cognitive Intelligence in Pre-trained Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show evidence for emergent cognitive abilities in Large Pre-trained Language Models (PLMs). Prior research into emergental cognitive abilities of PLMs has been path-independent to model training. |
| Approach: | They use four task categories to examine the alignment of ten popular families of PLMs and evaluate their performance to the developmental trajectories of children's thinking. |
| Outcome: | The results show that the models are more aligned to children's thinking than previous studies. |
DischargeSim: A Simulation Benchmark for Educational Doctor–Patient Communication at Discharge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Discharge communication is a critical yet underexplored component of patient care, where the goal shifts from diagnosis to education. |
| Approach: | They propose a benchmark that evaluates large language models’ ability to act as personalized discharge educators. |
| Outcome: | Experiments with 18 LLMs show that model size does not always yield better education outcomes, highlighting trade-offs in strategy use and content prioritization. |
Multitask Instruction-based Prompting for Fallacy Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Fallacies are used as seemingly valid arguments to support a position and persuade the audience about its validity. |
| Approach: | They propose to use instruction-based prompting to recognize 28 unique fallacies across datasets . they also analyze the effect of model size and prompt choice on model performance . |
| Outcome: | The proposed approach can recognize 28 unique fallacies across domains and genres. |
GenPTQ: Green Post-Training Quantization for Large-Scale ASR Models with Mixed-Precision Bit Allocation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large-scale models with high memory and computation demands face significant challenges for deployment. |
| Approach: | They propose a weight-only quantization method that optimizes for weights and QAT to restore accuracy. |
| Outcome: | The proposed method achieves 89.1% model size reduction with only 0.8% increase in WER and completes optimization in just 15 seconds. |
Pruning Foundation Models for High Accuracy without Retraining (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Despite the superior performance of foundation models, it is challenging to deploy large language models in practical applications due to their massive parameters and computations. |
| Approach: | They propose a pruning algorithm to prune LLMs in one-shot without retraining . they propose retrainable pruning algorithms to prune multiple weights in LLM . |
| Outcome: | The proposed pruning methods perform better than baseline pruning methods on sparse and unstructured sparsity models. |
Making Large Language Models Efficient Dense Retrievers (2026.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that fine-tuning large language models for dense retrieval yields strong performance, but their substantial parameter counts make them computationally inefficient. |
| Approach: | They propose a framework for developing efficient retrievers that performs coarse-to-fine compression through a coarse-grained coarse-tuning strategy. |
| Outcome: | The proposed framework reduces model size and inference cost while preserving performance of full-size models. |
GreenKGC: A Lightweight Knowledge Graph Completion Method (2023.acl-long)
Copied to clipboard
| Challenge: | Knowledge graph completion (KGC) aims to discover missing relationships in knowledge graphs (KGs). |
| Approach: | They propose a modularized knowledge graph completion solution that learns embeddings for entities and relations through a score function. |
| Outcome: | Experimental results show that GreenKGC outperforms SOTA methods in low dimensions and even better against high-dimensional models with a much smaller model size. |
CRaSh: Clustering, Removing, and Sharing Enhance Fine-tuning without Full Large Language Model (2023.emnlp-main)
Copied to clipboard
| Challenge: | Instruction tuning is an effective way of aligning large language models with private instruction data. |
| Approach: | They propose a training-free strategy to derive improved emulators from LLMs by using Offsite-Tuning (OFT) they propose CRaSh, which transfers transformer blocks between centralized LLM and downstream emulators . |
| Outcome: | The proposed technique boosts performance of large language models with billions of parameters. |
MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition (2024.findings-emnlp)
Copied to clipboard
Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, Bo Yuan
| Challenge: | emergence of Mixture of Experts (MoE) LLMs has significantly advanced the development of language models. |
| Approach: | They propose a two-stage compression method tailored for Mixture of Experts to reduce the model size and decrease the computational cost. |
| Outcome: | The proposed method reduces model size and improves inference efficiency while maintaining performance in various zero-shot tasks. |
Mapping Brains with Language Models: A Survey (2023.findings-acl)
Copied to clipboard
| Challenge: | accumulated evidence for brain and language model activations remains ambiguous, but correlations with model size and quality provide grounds for cautious optimism. |
| Approach: | They examine the evidence accumulated by 30 studies spanning 10 datasets and 8 metrics to determine whether there is any overlap between brain and language model activations. |
| Outcome: | The findings suggest that representations extracted from NLP models can (partially) explain the signal found in neural data. |
Intriguing Properties of Compression on Multilingual Models (2022.emnlp-main)
Copied to clipboard
Kelechi Ogueji, Orevaoghene Ahia, Gbemileke Onilude, Sebastian Gehrmann, Sara Hooker, Julia Kreutzer
| Challenge: | Multilingual models are dependent on scaling to generalize to a growing number of languages . compression techniques can have disparate effects on model performance for low-resource languages if used sparsely . |
| Approach: | They propose to characterize the impact of sparsifying multilingual pre-trained language models during fine-tuning. |
| Outcome: | The proposed framework characterizes the impact of sparsifying multilingual pre-trained language models during fine-tuning. |
Understanding and Overcoming the Challenges of Efficient Transformer Quantization (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in transformer quantization have shown remarkable improvement in many Natural Language Processing tasks and beyond. |
| Approach: | They propose a novel quantization scheme for transformers that can be quantized to ultra-low bit-widths, leading to significant memory savings with a minimum accuracy loss. |
| Outcome: | The proposed methods achieve state-of-the-art results on the GLUE benchmark using BERT, while preserving memory and accuracy. |
Vector-Vector-Matrix Architecture: A Novel Hardware-Aware Framework for Low-Latency Inference in NLP Applications (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to improve accuracy of neural networks are slow due to computational complexity. |
| Approach: | They propose a vector-vector-matrix architecture which greatly reduces latency at inference time for NLP applications by a factor of four. |
| Outcome: | The proposed framework reduces the latency of sequence-to-sequence and Transformer models used for NMT by a factor of four. |
From Text to Source: Results in Detecting Large Language Model-Generated Content (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) generate human-like text, but have ethical and misuse concerns. |
| Approach: | They evaluate whether a classifier trained to distinguish between source and target LLMs can detect text from an LLM without further training. |
| Outcome: | The proposed method detects text from target LLMs without further training. |
ICLEval: Evaluating In-Context Learning Ability of Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing evaluation frameworks focus on language abilities and knowledge, often overlooking the assessment of ICL ability. |
| Approach: | They propose to evaluate the ICL ability of Large Language Models (LLMs) using the ICLEval benchmark. |
| Outcome: | The proposed benchmark demonstrates that ICL ability is universally present in different LLMs and model size is not the sole determinant of ICL efficacy. |
Eeyore: Realistic Depression Simulation via Expert-in-the-Loop Supervised and Preference Optimization (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have been explored for mental healthcare training and therapy client simulation, but they fail to authentically capture diverse client traits and psychological conditions. |
| Approach: | They propose an 8B model optimized for realistic depression simulation with expert input at every stage. |
| Outcome: | The model outperforms GPT-4o in linguistic authenticity and profile adherence. |
Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Recent studies suggest that large language models can transfer skills learned in one language to others, but internal mechanisms behind this ability remain unclear. |
| Approach: | They find that LLMs map semantically identical inputs from different languages into a common semantic latent space that allows for consistent processing across languages. |
| Outcome: | The findings highlight the structural evolution of multilingual models during training and scaling up. |
Rethinking Task-Specific Knowledge Distillation: Contextualized Corpus as Better Textbook (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for knowledge distillation use a two-stage paradigm: general distillation with a task-agnostic general corpus and task-specific distillation using augmented task- specific corpus. |
| Approach: | They propose a contextualized corpus that contextualizes task corpus with large-scale general corpus through relevance-based text retrieval to improve student learning. |
| Outcome: | The proposed model improves on the GLUE benchmark and shows that it is better than generalized corpus and augmented task-specific corpus. |
How Well Can BERT Learn the Grammar of an Agglutinative and Flexible-Order Language? The Case of Basque. (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural Language Models (NLMs) have demonstrated effectiveness in acquiring skills related to human language use. |
| Approach: | They hypothesize that languages with complex grammar present substantial challenges during the pre-training phase . they constructed a test set that measures grammatical knowledge of BERT models trained under various pre-training configurations using corpus size, model size, number of epochs, and lemmatization. |
| Outcome: | The proposed model is based on a student-based minimal pairs test set with a grammatically correct and an incorrect sentence. |
MobileSpeech: A Fast and High-Fidelity Framework for Mobile Zero-Shot Text-to-Speech (2024.acl-long)
Copied to clipboard
| Challenge: | Existing zero-shot text-to-speech systems require a few seconds of unseen speaker voice prompts to generate high-quality voices. |
| Approach: | They propose a zero-shot text-to-speech system based on mobile devices . they use a discrete speech codec to integrate hierarchical information from the codec . |
| Outcome: | The proposed system achieves RTF of 0.09 on a single A100 GPU and has been successfully deployed on mobile devices. |
Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models are contaminated with data from pretraining and finetuning tasks. |
| Approach: | They perform extensive analysis on the factors that affect model memorization and generalization, such as model size, problem difficulty, and question length. |
| Outcome: | The results show that models perform better on the subset of the benchmarks where similar solutions are seen during training. |
AUTOSUMM: Automatic Model Creation for Text Summarization (2021.emnlp-main)
Copied to clipboard
Sharmila Reddy Nangi, Atharv Tyagi, Jay Mundra, Sagnik Mukherjee, Raj Snehal, Niyati Chhaya, Aparna Garimella
| Challenge: | Recent efforts to develop deep learning models for text generation tasks are challenging for non-experts. |
| Approach: | They propose methods to automatically create deep learning models for extractive and abstractive summarization tasks using large language models. |
| Outcome: | The proposed methods achieve near state-of-the-art performance on a range of datasets. |
Learning Language-Specific Layers for Multilingual Machine Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Multilingual Machine Translation (MNMT) is a promising new approach to improve translation quality between non-English languages. |
| Approach: | They propose a language-specific transformer layer to increase model capacity while keeping computation and parameters constant. |
| Outcome: | The proposed approach improves translation quality by 1.3 chrF (1.5 spBLEU) over not using LSLs on a separate decoder architecture. |
Fair or Framed? Political Bias in News Articles Generated by LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent Large Language Models (LLMs) have garnered significant attention for applications like news generation and opinion analysis. |
| Approach: | They analyze 10,850 articles and analyze their publicViews dataset to find left-leaning bias persists in generation tasks. |
| Outcome: | The proposed model size and the PublicViews dataset show that left-leaning bias persists in generation tasks and neutral content remains rare even under balanced opinion settings. |
VE-KD: Vocabulary-Expansion Knowledge-Distillation for Training Smaller Domain-Specific Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | VE-KD is a method that balances knowledge distillation and vocabulary expansion with the aim of training efficient domain-specific language models. |
| Approach: | They propose a method that balances knowledge distillation and vocabulary expansion with the aim of training efficient domain-specific language models. |
| Outcome: | VE-KD outperforms DistilBERT and Adapt-and-Distill in biomedical domain tasks . compared with other methods, it outperformed Distilbert and adapted-and distill . |
Multi-Docker-Eval: A ‘Shovel of the Gold Rush’ Benchmark on Automatic Environment Building for Software Engineering (2026.findings-acl)
Copied to clipboard
| Challenge: | Automated environment configuration is a critical bottleneck in scaling software engineering (SWE) automation. |
| Approach: | They propose a reliable evaluation standard for automated environment configuration for 40 real-world repositories spanning 9 programming languages. |
| Outcome: | The proposed benchmark includes 40 real-world repositories spanning 9 programming languages and measures success in achieving executable states and efficiency under realistic constraints. |
Less Is More? Examining Fairness in Pruned Large Language Models for Summarising Opinions (2025.emnlp-main)
Copied to clipboard
| Challenge: | reducing the size of LLMs through post-training pruning has been studied, but its impact on model fairness remains unexplored. |
| Approach: | They propose a pruning method that removes parameters that are redundant for input processing but influential in output generation. |
| Outcome: | The proposed pruning method can maintain or improve fairness across models and tasks where existing methods have limitations. |
Enhancing Scalability of Pre-trained Language Models via Efficient Parameter Sharing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to scale pre-trained language models to a deeper model depth share all parameters or use extra blocks. |
| Approach: | They propose a parameter-efficient approach to scaling pre-trained language models to a deeper model depth using matrix product operator. |
| Outcome: | The proposed model scales pre-trained language models to a deeper model depth by 4x and achieves 0.1 points higher than BERT-large for GLUE score. |
Group-SAE: Efficient Training of Sparse Autoencoders for Large Language Models via Layer Groups (2025.emnlp-main)
Copied to clipboard
| Challenge: | Sparse Autoencoders (SAEs) are a promising unsupervised approach for understanding the representations of layers of Large Language Models (LLMs). |
| Approach: | They propose a method that groups similar models and trains a single SAE per group based on representational similarity across layers. |
| Outcome: | Experiments on Pythia family models show that the proposed method significantly accelerates training with minimal impact on reconstruction quality and comparable downstream task performance and interpretability over baseline SAEs trained layer by layer. |
On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in language and vision assistants have showcased impressive capabilities but suffer from a lack of transparency, limiting broader research and reproducibility. |
| Approach: | They propose to redefine the design of vision-language models by identifying key components and creating efficient models with constrained inference costs. |
| Outcome: | The proposed models achieve significant improvements in inference throughput while maintaining high performance. |
Discrete Minds in a Continuous World: Do Language Models Know Time Passes? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models excel at temporal reasoning tasks, but their ability to perceive the passage of time remains unexplored. |
| Approach: | They propose a Token-Time Hypothesis to test whether LLMs perceive the passage of time . they also propose an interactive navigation challenge to examine how LLM responds to time pressure . |
| Outcome: | The proposed model can map discrete token counts to wall-clock time and validate this through a dialogue duration judgment task. |
A linguistically-motivated evaluation methodology for unraveling model’s abilities in reading comprehension tasks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing models fail for linguistic characteristics of input examples, despite the impressive quantity of scientific studies dedicated to them, the capabilities, limitations, and risks of these models remain largely unknown. |
| Approach: | They propose to use semantic frame annotation to characterize examples by a small number of complexity factors to account for model’s difficulty. |
| Outcome: | The proposed evaluation methodology is based on the intuition that certain examples consistently yield lower scores regardless of model size or architecture. |
On the Scaling Laws of Geographical Representation in Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Language models embed geographical information in their hidden representations, but larger models cannot mitigate this bias. |
| Approach: | They propose to extend this finding to Large Language Models by observing how geographical knowledge evolves when scaling language models. |
| Outcome: | The proposed model scales consistently with increasing model size, but smaller models cannot mitigate geographic bias inherent in training data. |
On the Way to Lossless Compression of Language Transformers: Exploring Cross-Domain Properties of Quantization (2024.lrec-main)
Copied to clipboard
Nikita Martynov, Aleksei Goncharov, Gleb Kumichev, Evgeniy Egorov, Stanislav Vladimirovich Pavlov, Mikhail Sergeevich Durinov, Aleksandr Sergeevich Zuev, Egor Anatolievich Filimonov
| Challenge: | Modern Natural Language Processing models have a huge capacity, but this makes it difficult to employ. |
| Approach: | They propose a method to quantize at least 95% of Transformer weights without access to task-specific data so the drop in performance does not exceed 0.02%. |
| Outcome: | The proposed method quantizes 95% of Transformer weights and corresponding activations to INT8 without access to task-specific data so the drop in performance does not exceed 0.02%. |
On Zero-Shot Counterspeech Generation by LLMs (2024.lrec-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are used in numerous NLP tasks, including counterspeech generation. |
| Approach: | They propose three different prompting strategies for generating different types of counterspeech and propose a set of prompting techniques for counterspeak generation. |
| Outcome: | The proposed prompting strategies improve the performance of the models for counterspeech generation in two datasets, but with high toxicity with increase in model size. |
Lemmatization of Polish Multi-word Expressions (2025.emnlp-main)
Copied to clipboard
| Challenge: | PolEval task is complicated by linguistic irregularities and historical factors. |
| Approach: | They propose to use fine-tuned plT5 and mT5 models to improve lemmatization of multi-word expressions and proper names in Polish instead of rule-based methods. |
| Outcome: | The proposed model achieved 86.23% AccCS, 89.43% accCI and 88.79% on a gold-standard dataset derived from Wikipedia . |
Language Models Resist Alignment: Evidence From Data Compression (2025.acl-long)
Copied to clipboard
Jiaming Ji, Kaile Wang, Tianyi Alex Qiu, Boyuan Chen, Jiayi Zhou, Changye Li, Hantao Lou, Josef Dai, Yunhuai Liu, Yaodong Yang
| Challenge: | Large language models (LLMs) may exhibit undesirable behaviors due to the inevitable biases and harmful content present in training. |
| Approach: | They propose to investigate the elasticity of large language models by examining their performance. |
| Outcome: | The proposed model performance declines rapidly before reverting to the pre-training distribution, the authors show . the proposed model weight and code are available at pku-lm-res ist-alignment.github.io. |
Pruning before Fine-tuning: A Retraining-free Compression Framework for Pre-trained Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Structured pruning is an effective technique for compressing pre-trained language models (PLMs), but it requires retraining, leading to additional computational overhead. |
| Approach: | They propose a task-specific pruning framework that prunes redundant modules of pre-trained language models before fine-tuning them. |
| Outcome: | The proposed pruning framework achieves higher performance on GLUE, SQUAD, WikiText-2, Wik-103, and PTB datasets while reducing the time required for fine-tuning. |
Revisiting Scaling Laws for Language Models: The Role of Data Quality and Training Strategies (2025.acl-long)
Copied to clipboard
| Challenge: | Existing scaling laws suggest augmenting model size and training data results in enhanced performance, but recent studies reveal deviations, particularly in large language models, where performance improvements decelerate—a phenomenon known as sub-scaling. |
| Approach: | They propose a sub-optimal scaling law that better predicts performance in sub-scaling regimes by examining data quality and training strategies. |
| Outcome: | The proposed scaling law better predicts performance in sub-scaling regimes, highlighting the importance of data quality and diversity. |
On the Robust Approximation of ASR Metrics (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for estimating speech recognition metrics depend on ground truth labels. |
| Approach: | They propose a label-free approach to approximating ASR performance metrics . they embed multimodal embeddings in a unified space for speech and transcription representations . |
| Outcome: | The proposed method outperforms baseline models on speech recognition benchmarks by 50%. |
SLlama: Parameter-Efficient Language Model Architecture for Enhanced Linguistic Competence Under Strict Data Constraints (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large-scale language models (LLMs) have shown remarkable performance across a wide array of tasks. |
| Approach: | They propose an architecture that preserves parameter efficiency of tied models without sacrificing representational benefits of untied embeddings. |
| Outcome: | The proposed architecture achieves a 31.72% improvement in linguistic knowledge acquisition over the baseline model. |
LegoMT2: Selective Asynchronous Sharded Data Parallel Training for Massive Neural Machine Translation (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to train a single model for massive languages have huge communication overheads and parameter interference. |
| Approach: | They propose an efficient training approach with an asymmetric multi-way model architecture for massive multilingual neural machine translation. |
| Outcome: | The proposed model is 16.2 faster than the distributed training method for M2M-100-12B while improving the translation performance by an average of 2.2 BLEU on Flores-101. |
Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training (2026.acl-long)
Copied to clipboard
Changhao Jiang, Ming Zhang, Yifei Cao, Junjie Ye, Xiaoran Fan, Shihan Dou, Zhiheng Xi, Jiajun Sun, Yi Dong, Yujiong Shen, Jingqi Tong, Baoyu Fan, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing methods to predict performance of large language models are lacking . authors propose a size-dependent mutual information predictor for closed-book question answering accuracy . |
| Approach: | They propose a size-dependent mutual information predictor that integrates knowledge frequency, knowledge specificity, and model size to forecast closed-book question answering accuracy. |
| Outcome: | The proposed method outperforms baseline models and achieves R2 > 0.7 in predicting QA accuracy without additional training. |
Brittle Minds, Fixable Activations: Understanding Belief Representations in Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Despite growing interest in Theory of Mind (ToM) tasks for evaluating language models, little is known about how LMs internally represent mental states of self and others. |
| Approach: | They propose to investigate how LMs internally represent mental states of self and others . |
| Outcome: | The proposed model size and finetuning significantly improve LMs’ internal representations of others’ beliefs, which are structured - not mere by-products of spurious correlations - yet brittle to prompt variations. |
Revisiting the Self-Consistency Challenges in Multi-Choice Question Formats for Large Language Model Evaluation (2024.lrec-main)
Copied to clipboard
| Challenge: | Multi-choice questions (MCQs) are a common method for assessing the world knowledge of large language models. |
| Approach: | They propose three knowledge-equivalent question variants to assess LLMs' world knowledge . they propose option position shuffle, option label replacement, and conversion to a True/False format . |
| Outcome: | The proposed questions are shuffle, label replacement, and True/False format. |
Efficient Hyperparameter Optimization for LLM Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing hyperparameter optimization methods are inefficient in reinforcement learning due to model scale and resource-intensive training cycles. |
| Approach: | They propose a hyperparameter optimization method that adapts both model size and training budget as fidelity. |
| Outcome: | The proposed method significantly improves the computational efficiency of each trial (up to 14.9) over existing HPO methods. |
Sequence-to-Sequence Language Models for Character and Emotion Detection in Dream Narratives (2024.lrec-main)
Copied to clipboard
| Challenge: | Sigmund Freud's interpretation of dreams has been central to understanding human consciousness for centuries. |
| Approach: | They propose to automate the annotation process by using a natural language framework . they evaluate the impact of model size, prediction order of characters, and consideration of proper names and character traits . |
| Outcome: | The proposed model performs better with a large language model while having 28 times fewer parameters. |
DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization (2025.acl-long)
Copied to clipboard
| Challenge: | Structured pruning reduces model size but often causes uneven degradation across domains, leading to biased performance. |
| Approach: | They propose a method that dynamically adjusts the data distribution during training to restore balanced performance across heterogeneous and multi-tasking data. |
| Outcome: | Experiments in monolingual and multilingual settings show that the proposed method surpasses similarly sized models in pruning and continued pretraining over perplexity, downstream tasks, and instruction tuning. |
MENTOR: Efficient Autoregressive Image Generation with Balanced Multimodal Control (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent text-to-image models achieve impressive visual quality but still face challenges in precise controllability, balancing multimodal inputs, and high training cost for multimodal image generation. |
| Approach: | They propose an autoregressive framework with a two-stage training paradigm for controllable multimodal image generation. |
| Outcome: | Extensive experiments on DreamBench++ and DreamBech show that the proposed framework achieves a strong balance between textual and visual guidance for controllable image generation. |
On the Cultural Anachronism and Temporal Reasoning in Vision Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Vision-Language Models (VLMs) are increasingly applied to cultural heritage materials. |
| Approach: | They propose a temporal anachronism benchmark to evaluate temporal reasoning on 1,600 Indian cultural artifacts. |
| Outcome: | The proposed model achieves only 58.7% accuracy on the best model, which is a significant performance gap across architectures and scales. |
Convergence and Divergence of Language Models under Different Random Seeds (2025.emnlp-main)
Copied to clipboard
| Challenge: | a large body of work has examined the training dynamics of language models. |
| Approach: | They investigate the convergence of language models (LMs) trained under different random seeds . they find that larger models reconverge faster in later training stages, while smaller models never actually reconverge. |
| Outcome: | The proposed model size and training checkpoints influence convergence of language models under different seeds. |
Lightweight and Faithful Visual Condition Checking in Behavior Trees via Expert-Regularized Reinforcement Learning (2026.acl-long)
Copied to clipboard
| Challenge: | Existing behavior trees are not suitable for high-dimensional perceptual inputs such as images or language. |
| Approach: | They propose a framework that leverages expert-regularized reinforcement learning to preserve semantic faithfulness while employing a factorized policy that aggregates sequential condition-node decisions into a single decision unit. |
| Outcome: | The proposed framework outperforms imitation learning and reinforcement learning but risks misalignment of condition nodes with intended semantics and poor credit assignment. |
GUIDE: Towards Scalable Advising for Research Ideas (2026.acl-long)
Copied to clipboard
| Challenge: | Existing systems that provide detailed, constructive feedback on academic papers struggle with review fidelity. |
| Approach: | They explore factors that underlie the development of robust advising systems . large language models have shown remarkable progress in tasks from text generation to code synthesis . |
| Outcome: | The proposed model outperforms general-purpose language models in acceptance rates for self-ranked top-30% submissions to ICLR 2025. |
WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for reinforcement learning (RL)-based agents struggle with long-horizon planning and strategy coherence. |
| Approach: | They propose a reinforcement learning framework that decouples planning and execution. |
| Outcome: | The proposed framework outperforms baseline and first-step RL frameworks on four benchmarks. |